Repository navigation
Conversation
Features: - Add Day and Week timelines for Dag run timing patterns - Group runs by Dag, state, time bucket, and weekday - Support Mean, Max, and Min duration aggregation - Show expected runs for scheduled Dags without matching runs - Add Dag run, tag, timetable type, and team filters Interactions: - Add pointer-anchored timeline zoom controls - Link timeline bars to their Dag or Dag run - Add readable tooltips for run details - Persist view, aggregation, limit, and filter preferences - Support overlapping runs in Day and Week layouts Data loading: - Add selectable limits from 200 runs through all runs - Paginate using the API-configured page size - Prevent gaps between paginated Dag run results - Limit Dag detail requests to expected-run markers Documentation and tests: - Document Time Schedule behavior in the UI guide - Add light and dark screenshots - Cover layouts, aggregation, filters, zoom, and pagination
boring-cyborg
Bot
added
area:translations
area:UI
|
Very cool! I could see a world where we could even show arrows between runs that are connected via partition keys. Or highlighting when Max active dag runs is being hit. Love the minute detailing. That's great for dags that run every minute! Just needs a label.
|
bbovenzi
left a comment
bbovenzi
left a comment
There was a problem hiding this comment.
This is a cool demo. But it definitely needs some work to "productionize" the code to be usable for deployments with any degree of scale.
Capture changes during partial author review: Data access and correctness: - Avoid browser request fan-out and client-side filtering as Dag and run counts grow. - Bound each view to 5,000 matching runs rather than allowing unbounded reads. - Calculate only the selected Day or Week aggregation with access controls and filters applied. Interaction quality: - Stream timeline batches so useful data appears before the complete result is ready. - Preserve displayed bars during zoom-driven refreshes while the final zoom level settles. - Keep planned Dag markers visible even when no Dag runs match the current view. Contract and verification: - Keep the private UI OpenAPI contract and generated client aligned with the endpoint. - Cover streaming, limits, filtering, aggregation, planned runs, and zoom behavior.
- Avoid MySQL's unsupported LIMIT subquery in an IN predicate. - Mark NDJSON reads as sequential and fix TimeSchedule type errors. - Confirm earlier UI import failures no longer affect asset builds.
Keep Time Schedule compatible with the shared FilterBar refactor merged from main. - Deleted Dag list filter imports caused Vite module-resolution failures. - The failure blocked React UI tests and CI jobs that build UI assets. - Reuse the shared filter state for Tag and Timetable Type filtering.
|
@bbovenzi Since b13f993, excluding merge commits, I made the following changes:
The current flow is: flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
The browser now receives only the filtered data needed for the selected view, while the backend controls the result size and aggregation cost. I initially intended this change to remain UI-only and did not plan to add a backend API. Because the review identified the scalability limitations of the original approach, I needed to first understand Airflow's backend API structure, streaming session lifecycle, authorization filters, and OpenAPI generation flow. I also spent additional time reviewing the AI-assisted code carefully, which is why the follow-up changes took about 5 days. The implementation is now focused on the production-scale concerns from the review while preserving the existing Airflow UI/API patterns. |
|
I missed these additional suggestions in my previous follow-up. I plan to address the following:
For the partition-key arrows, I understand this may include dependencies between Dag runs, such as those created through I’m happy to implement this with more specific guidance on which relationships to show and how they should be laid out. With the current context, I’m hesitant to add a potentially confusing visualization. Drafted-by: GPT-5 |
Integrate Time Schedule into the main Dashboard stats and improve control usability, accessibility, and documentation: Dashboard and routing: - Add a Time Schedule `StatsCard` with `FiCalendar` icon to the Dashboard `Stats` section linking to `/time_schedule`. - Clean up redundant layout imports in `DagsLayout.tsx`. Control and interaction refinements: - Switch the Day/Week view switcher to Chakra UI `Tabs` inside `TimeScheduleControls`. - Add `ControlHelp` info tooltips (`FiInfo`) explaining Zoom controls, Dag run limit, and Duration aggregation modes. - Refactor `TimelineBar` and `TimelineTooltip` with improved accessibility, state styling, and shared tooltip props in `constants.ts`. - Extract common layout constants into `constants.ts` and refine date formatting utilities. Documentation and tests: - Add unit tests for `TimelineBar` and `timelineUtils`. - Update `ui.rst` and capture updated light/dark mode screenshots for Time Schedule views and Dashboard stats.
|
@bbovenzi I've addressed the remaining Time Schedule suggestions and wrapped up the final refinements. Dashboard entry pointTime Schedule is no longer surfaced from the Dags area. It is now available from the Stats section on the Dashboard home page. 
State icons on timeline barsTimeline bars now include the shared Airflow state icons alongside their labels, so Dag run state is distinguishable without relying on color alone. The same icon treatment is also used in the bar tooltip. 
Labels and tooltips
Day and Week as view tabsDay and Week are now presented as tabs rather than as a simple setting, matching the interaction pattern used for calendar-style view changes elsewhere in the UI. 
DocumentaionI also updated the UI documentation and screenshots to reflect the Dashboard entry point and the current Time Schedule controls. |
bbovenzi
left a comment
bbovenzi
left a comment
There was a problem hiding this comment.
Please have your agent check for pre-existing code instead of inventing its own practices.
I still don't think our endpoint is structured the right way to make this scale properly. . Today we are fetching every single dag with no pagination and no default filter. We have an API roundtrip for every zoom level change but have no auto refresh. We have no autorefresh support. The UI has no virtualization either. Teams can have thousands of dags.
Feel free to start a chat on slack. There is a lot of API design and UX design to discuss.
| <Tabs.Root | ||
| data-testid="time-schedule-view-mode" | ||
| onValueChange={({ value }) => { | ||
| if (value === "day" || value === "week") { | ||
| onViewModeChange(value); | ||
| } | ||
| }} | ||
| size="sm" | ||
| value={viewMode} | ||
| variant="line" | ||
| > | ||
| <Tabs.List | ||
| bg="bg.muted" | ||
| borderColor="border.subtle" | ||
| borderRadius="md" | ||
| borderWidth="1px" | ||
| gap={1} | ||
| p={1} | ||
| > | ||
| {(["day", "week"] as const).map((mode) => ( | ||
| <Tabs.Trigger | ||
| _hover={{ bg: "bg.subtle", color: "fg" }} | ||
| _selected={{ bg: "brand.muted", color: "fg", fontWeight: "bold" }} | ||
| borderRadius="sm" | ||
| color="fg.muted" | ||
| fontSize="md" | ||
| fontWeight="medium" | ||
| height="32px" | ||
| justifyContent="center" | ||
| key={mode} | ||
| value={mode} | ||
| width="88px" | ||
| > | ||
| {translate(`timeSchedule.${mode}`)} | ||
| </Tabs.Trigger> | ||
| ))} | ||
| </Tabs.List> | ||
| </Tabs.Root> |
There was a problem hiding this comment.
We can use our pre-existing ButtonGroupToggle component here instead.
There was a problem hiding this comment.
import { ButtonGroupToggle } from "src/system-components";| <Tabs.Root | |
| data-testid="time-schedule-view-mode" | |
| onValueChange={({ value }) => { | |
| if (value === "day" || value === "week") { | |
| onViewModeChange(value); | |
| } | |
| }} | |
| size="sm" | |
| value={viewMode} | |
| variant="line" | |
| > | |
| <Tabs.List | |
| bg="bg.muted" | |
| borderColor="border.subtle" | |
| borderRadius="md" | |
| borderWidth="1px" | |
| gap={1} | |
| p={1} | |
| > | |
| {(["day", "week"] as const).map((mode) => ( | |
| <Tabs.Trigger | |
| _hover={{ bg: "bg.subtle", color: "fg" }} | |
| _selected={{ bg: "brand.muted", color: "fg", fontWeight: "bold" }} | |
| borderRadius="sm" | |
| color="fg.muted" | |
| fontSize="md" | |
| fontWeight="medium" | |
| height="32px" | |
| justifyContent="center" | |
| key={mode} | |
| value={mode} | |
| width="88px" | |
| > | |
| {translate(`timeSchedule.${mode}`)} | |
| </Tabs.Trigger> | |
| ))} | |
| </Tabs.List> | |
| </Tabs.Root> | |
| <ButtonGroupToggle<ViewMode> | |
| data-testid="time-schedule-view-mode" | |
| marginStart="auto" | |
| onChange={onViewModeChange} | |
| options={[ | |
| { label: translate("timeSchedule.day"), value: "day" }, | |
| { label: translate("timeSchedule.week"), value: "week" }, | |
| ]} | |
| size="sm" | |
| value={viewMode} | |
| /> |
| export const formatDurationLabel = (durationMs: number) => { | ||
| if (durationMs <= 0) { | ||
| return ""; | ||
| } | ||
| const seconds = Math.max(1, Math.round(durationMs / 1000)); | ||
|
|
||
| if (seconds < 60) { | ||
| return `${seconds}s`; | ||
| } | ||
| const minutes = Math.floor(seconds / 60); | ||
|
|
||
| if (minutes < 60) { | ||
| return `${minutes}m`; | ||
| } | ||
|
|
||
| return minutes % 60 > 0 ? `${Math.floor(minutes / 60)}h ${minutes % 60}m` : `${Math.floor(minutes / 60)}h`; | ||
| }; |
There was a problem hiding this comment.
Let's not create a new formatDuration function and use our existing one in datetimeUtils.ts
There was a problem hiding this comment.
getTimelineDurationSeconds keeps bar labels compact while reusing Airflow’s shared renderDuration formatter: seconds are retained for sub-minute runs (50s) but omitted for longer runs (6m, 1h 2m).
import { renderDuration } from "src/utils/datetimeUtils";| export const formatDurationLabel = (durationMs: number) => { | |
| if (durationMs <= 0) { | |
| return ""; | |
| } | |
| const seconds = Math.max(1, Math.round(durationMs / 1000)); | |
| if (seconds < 60) { | |
| return `${seconds}s`; | |
| } | |
| const minutes = Math.floor(seconds / 60); | |
| if (minutes < 60) { | |
| return `${minutes}m`; | |
| } | |
| return minutes % 60 > 0 ? `${Math.floor(minutes / 60)}h ${minutes % 60}m` : `${Math.floor(minutes / 60)}h`; | |
| }; | |
| export const getTimelineDurationSeconds = (durationMs: number) => | |
| durationMs >= 60_000 ? Math.floor(durationMs / 60_000) * 60 : durationMs / 1000; | |
| export const getTimelineBarMinimumWidth = (durationMs: number, locale?: string) => | |
| STATE_ICON_AND_SPACING_WIDTH_PX + | |
| (durationMs > 0 ? (renderDuration(getTimelineDurationSeconds(durationMs), locale)?.length ?? 0) : 0) * | |
| DURATION_CHARACTER_WIDTH_PX; |
| */ | ||
| import dayjs from "dayjs"; | ||
| import timezone from "dayjs/plugin/timezone"; | ||
| import utc from "dayjs/plugin/utc"; | ||
|
|
||
| dayjs.extend(utc); | ||
| dayjs.extend(timezone); | ||
|
|
||
| export { default as dayjs } from "dayjs"; |
| const formatTime = (datetime: string | null, selectedTimezone: string) => | ||
| datetime === null ? "—" : dayjs(datetime).tz(selectedTimezone).format("HH:mm"); | ||
|
|
||
| export const TimelineTooltip = ({ item, selectedTimezone }: TimelineTooltipProps) => { |
There was a problem hiding this comment.
All these tooltips are styled different from the rest of the UI. Let's please make this more consistent.
There was a problem hiding this comment.
airflow-core/src/airflow/ui/src/pages/TimeSchedule/TimelineBar.tsx
- <Tooltip content={renderTooltip(item)} contentProps={TIMELINE_TOOLTIP_CONTENT_PROPS}>
+ <Tooltip content={renderTooltip(item)}>| const formatTime = (datetime: string | null, selectedTimezone: string) => | ||
| datetime === null ? "—" : dayjs(datetime).tz(selectedTimezone).format("HH:mm"); |
There was a problem hiding this comment.
We already have a format date and format time functions.
There was a problem hiding this comment.
import { formatDate } from "src/utils/datetimeUtils";| const formatTime = (datetime: string | null, selectedTimezone: string) => | |
| datetime === null ? "—" : dayjs(datetime).tz(selectedTimezone).format("HH:mm"); | |
| const formatTime = (datetime: string | null, selectedTimezone: string) => | |
| datetime === null ? "—" : formatDate(datetime, selectedTimezone, "HH:mm"); |
| state: item.state, | ||
| }); | ||
|
|
||
| export const useTimeScheduleData = ({ |
There was a problem hiding this comment.
A lot of this is similar to useGridTiSummariesStream. Let's try to extract out some reusable functions.
There was a problem hiding this comment.
That + react query should also make it easier to add auto refresh. We should probably think about how to stream this info with auto refresh in a lightweight manner instesd of always appending info.
There was a problem hiding this comment.
I switched to React Query’s streamedQuery and Airflow’s existing useAutoRefresh:
import { experimental_streamedQuery as streamedQuery, useQuery } from "@tanstack/react-query";
import { useAutoRefresh } from "src/utils";export const useTimeScheduleData = (streamQuery: string) => {
const refetchInterval = useAutoRefresh({});
const nonZoomQuery = new URLSearchParams(streamQuery);
nonZoomQuery.delete("time_scale");
const nonZoomStreamQuery = nonZoomQuery.toString();
const query = useQuery({
placeholderData: (previousData, previousQuery) =>
previousQuery?.queryKey[1] === nonZoomStreamQuery ? previousData : undefined,
queryFn: streamedQuery<TimeScheduleBatch>({
refetchMode: "replace",
streamFn: ({ signal }) => streamTimeSchedule(streamQuery, signal),
}),
queryKey: ["time-schedule", nonZoomStreamQuery, streamQuery],
refetchInterval,
refetchIntervalInBackground: false,
retry: false,
});
...I’ll follow up separately with questions about further improvements.
| const buildTimelineItem = (item: TimeScheduleItem): TimelineItem => ({ | ||
| dagId: item.dag_id, | ||
| dagRunId: item.dag_run_id, | ||
| durationMs: item.duration_ms, | ||
| endDate: item.end_date, | ||
| isPlaceholder: item.is_placeholder, | ||
| isPlanned: item.is_planned, | ||
| isTimeScheduled: item.is_time_scheduled, | ||
| label: item.label, | ||
| runCount: item.run_count, | ||
| startDate: item.start_date, | ||
| state: item.state, | ||
| }); |
There was a problem hiding this comment.
Let's just use the API's snake_casing. No need to maintain our own camelCasing conversion
There was a problem hiding this comment.
Removed buildTimelineItem and now use the generated API types directly, preserving snake_case fields.
deleted:
const buildTimelineItem = (item: TimeScheduleItem): TimelineItem => ({ ... });
useEffect(() => {
...
const applyBatch = (batch: TimeScheduleBatch, shouldReplaceTimelineItems: boolean) => {
const batchItems = batch.items.map(buildTimelineItem);
...added:
export const useTimeScheduleData = (streamQuery: string) => {
...
const query = useQuery({ ... });
return {
dagRunCount: query.data?.reduce((count, batch) => count + batch.dag_run_count, 0) ?? 0,
error: query.error,
isLoading: query.isPending,
timelineItems: query.data?.flatMap((batch) => batch.items) ?? [],
};There was a problem hiding this comment.
This file should should be moved to src/queries
There was a problem hiding this comment.
Moved useTimeScheduleData.ts from src/pages/TimeSchedule to src/queries and updated the import.
airflow-core/src/airflow/ui/src/queries
├── useTimeScheduleData.test.tsx
└── useTimeScheduleData.ts| const [error, setError] = useState<Error>(); | ||
| const [isLoading, setIsLoading] = useState(true); |
There was a problem hiding this comment.
We should use react query at least a little bit instead of handling loading and error on our own.
There was a problem hiding this comment.
Replaced the manual loading and error state with react query’s query.isPending and query.error.
export const useTimeScheduleData = (streamQuery: string) => {
...
const query = useQuery({ ... });
return {
...
error: query.error,
isLoading: query.isPending,
...
};| is_placeholder: bool | ||
| is_planned: bool | ||
| is_time_scheduled: bool | ||
| label: str |
There was a problem hiding this comment.
Let's not rename dag_display_name
There was a problem hiding this comment.
class TimeScheduleItem(BaseModel):
"""An aggregated bar in the Time Schedule UI."""
dag_id: str
dag_run_id: str
duration_ms: float
end_date: datetime | None
is_placeholder: bool
is_planned: bool
is_time_scheduled: bool
- label: str
+ dag_display_name: str
run_count: int
start_date: datetime | None
state: DagRunState | Literal["placeholder", "planned"]- Use React Query streaming and auto-refresh for Time Schedule data. - Apply scheduled-only filtering through the shared FilterPill and API. - Clarify duration labels and keep bar widths consistent with their labels. - Align tooltips, zoom controls, and Day/Week tabs with Airflow UI patterns. - Update routing, generated API types, documentation, translations, and tests.
- Remove Time Schedule from the Dag stats cards - Add a separate linked card beside Stats on the Dashboard - Update the UI documentation and add card navigation tests
- Limit displayed Dags to those represented by the selected runs - Virtualize Day view rows for larger run limits - Link aggregated bars to filtered Dag Runs lists - Add Start Time ranges and multi-select Start Weekday filters
- Provide timezone context required by the new time filters - Format the Dag run limit counter according to locale conventions - Align the limit option test with formatted number output
This branch has not been deployed
2 participants
Add a Time Schedule tab for comparing Dag start times and durations across an Airflow environment.
Features:
This is a UI-only change. It reuses existing endpoints and adds no schema or backend API changes.
Earlier work in #68532 and #68547 informed the time-bucket design. The view placement follows #45497, while bounded loading follows the work in #47900, #60241, and #65388.
Related: #22001
Screenshots:
Videos:
time_schedule-day-zoom_in_out.mp4
time_schedule-week-zoom_in_out.mp4
time_schedule-day-bar_link.mp4
time_schedule-week-display_n_dags.mp4
time_schedule-day-aggregation.mp4
Was generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-5.6 Sol) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.